[14.0][IMP] mail_tracking: add job to delete old tracking records #1430

henrybackman · 2024-08-08T08:37:31Z

Records in mail_tracking_email table grow over time and increase the database size, which slows down backup and restore operations. (For example, in one customer db the mail_tracking_email table takes almost 2GB)

Added autovacuum to mail_tracking_email that removes by default records over 6 months old.
The deletion can be enabled with a configuration variable.
Due to possibly a large number of records to be deleted on first run, set a default limit of 5000 per run.

mail_tracking/models/mail_tracking_email.py

mail_tracking/models/res_config_settings.py

pedrobaeza · 2024-08-08T09:30:24Z

Do you have some metrics about performance degradation that leads to this action?

sebalix · 2024-08-08T11:34:39Z

@pedrobaeza this action is here to reduce the DB size (to speed up backup and restore operations mainly), currently we have a DB with the mail_tracking_value taking almost 2GB (in the top 10 of biggest table in this DB, 8th position, 1st being mail_message).
About performance I checked but there isn't a real impact here (while this similar PR about notifications is reducing a lot the time needed to open the Odoo JS client: #1340)

pedrobaeza · 2024-08-08T11:37:35Z

OK, thanks for the information. It's good to add this information initially (and include it in the commit message) for avoiding to guess what is the improvement.

It would be interesting to check if there's a way to optimize the DB schema of this table to reduce this size as well. Maybe there's a related stored or similar fields that increase the size a lot.

henrybackman · 2024-08-08T11:54:43Z

Thank you @pedrobaeza for checking. I don't have anything to add to @sebalix comment but I added the relevant details to the description 👍

henrybackman · 2024-08-09T11:19:38Z

Test added to the tests/__init__.py and improved wording in the settings description

mail_tracking/tests/test_gc_mail_tracking_email.py

simahawk · 2024-08-09T12:44:08Z

@henrybackman when you push again pls report the description in the commit msg.
Note: if you do that before opening a PR you'll get the PR description for free 😉

mail_tracking/models/mail_tracking_email.py

sebalix · 2024-08-13T14:23:08Z

mail_tracking/models/mail_tracking_email.py

+        records_to_delete = self.search(domain, limit=limit)
+        _logger.debug("Deleting %s mail.tracking.email records", len(records_to_delete))
+        return records_to_delete.unlink()


After testing this on a customer DB, it triggers the following error (linked to related fields):

Traceback (most recent call last): File "/odoo/src/odoo/addons/base/models/ir_autovacuum.py", line 38, in _run_vacuum_cleaner func(model) File "/odoo/external-src/social/mail_tracking/models/mail_tracking_email.py", line 503, in _gc_mail_tracking_email records_to_delete.unlink() File "/odoo/external-src/connector/component_event/models/base.py", line 118, in unlink result = super(Base, self).unlink() File "/odoo/src/odoo/models.py", line 3501, in unlink self.flush() File "/odoo/src/odoo/models.py", line 5472, in flush self.recompute() File "/odoo/src/odoo/models.py", line 5931, in recompute process(field) File "/odoo/src/odoo/models.py", line 5918, in process field.recompute(existing) File "/odoo/src/odoo/fields.py", line 1178, in recompute self.compute_value(record) File "/odoo/src/odoo/fields.py", line 1198, in compute_value records._compute_field_value(self) File "/odoo/src/odoo/models.py", line 4079, in _compute_field_value odoo.fields.determine(field.compute, self) File "/odoo/src/odoo/fields.py", line 85, in determine return needle(records, *args) File "/odoo/src/odoo/fields.py", line 574, in _compute_related record[self.name] = self._process_related(value[self.related_field.name]) File "/odoo/src/odoo/models.py", line 5702, in __getitem__ return self._fields[key].__get__(self, type(self)) File "/odoo/src/odoo/fields.py", line 2495, in __get__ return super().__get__(records, owner) File "/odoo/src/odoo/fields.py", line 1025, in __get__ _("(Record: %s, User: %s)") % (record, env.uid), odoo.exceptions.MissingError: Record does not exist or has been deleted.

I tried different things but only a raw DELETE query is working while being simple:

Suggested change

records_to_delete = self.search(domain, limit=limit)

_logger.debug("Deleting %s mail.tracking.email records", len(records_to_delete))

return records_to_delete.unlink()

records_to_delete = self.search(domain, limit=limit).exists()

if records_to_delete:

_logger.info("Deleting %s mail.tracking.email records", len(records_to_delete))

self.flush()

query = "DELETE FROM mail_tracking_email WHERE id IN %s"

args = (tuple(records_to_delete.ids),)

self.env.cr.execute(query, args)

self.invalidate_cache()

mail_tracking/models/mail_tracking_email.py

Add autovacuum to mail_tracking_email that removes old records based on new configuration variable mail_tracking_email_max_age_days. Due to possibly a large number of records to be deleted on first run, set a default limit of 5000 per run.

simahawk · 2024-08-26T14:24:06Z

@pedrobaeza good for you?

pedrobaeza

/ocabot merge minor

OCA-git-bot · 2024-08-26T14:28:03Z

This PR has the approved label and has been created more than 5 days ago. It should therefore be ready to merge by a maintainer (or a PSC member if the concerned addon has no declared maintainer). 🤖

OCA-git-bot · 2024-08-26T14:28:05Z

What a great day to merge this nice PR. Let's do it!
Prepared branch 14.0-ocabot-merge-pr-1430-by-pedrobaeza-bump-minor, awaiting test results.

OCA-git-bot · 2024-08-26T14:50:53Z

Congratulations, your PR was merged at 3bd12c2. Thanks a lot for contributing to OCA. ❤️

henrybackman force-pushed the 14.0-mail-tracking-email-deletion-job branch from 37b4f95 to 4e36e51 Compare August 8, 2024 08:44

sebalix reviewed Aug 8, 2024

View reviewed changes

mail_tracking/models/mail_tracking_email.py Outdated Show resolved Hide resolved

mail_tracking/models/res_config_settings.py Outdated Show resolved Hide resolved

henrybackman force-pushed the 14.0-mail-tracking-email-deletion-job branch from 34e286e to 36b36cf Compare August 8, 2024 11:48

henrybackman force-pushed the 14.0-mail-tracking-email-deletion-job branch from 36b36cf to dc9ef3c Compare August 8, 2024 11:56

henrybackman requested a review from sebalix August 8, 2024 12:42

sebalix approved these changes Aug 8, 2024

View reviewed changes

henrybackman force-pushed the 14.0-mail-tracking-email-deletion-job branch from dc9ef3c to 7e95078 Compare August 9, 2024 11:18

simahawk reviewed Aug 9, 2024

View reviewed changes

mail_tracking/tests/test_gc_mail_tracking_email.py Outdated Show resolved Hide resolved

henrybackman force-pushed the 14.0-mail-tracking-email-deletion-job branch from 7e95078 to 3fa48e9 Compare August 9, 2024 13:51

sebalix reviewed Aug 13, 2024

View reviewed changes

mail_tracking/models/mail_tracking_email.py Outdated Show resolved Hide resolved

sebalix suggested changes Aug 13, 2024

View reviewed changes

henrybackman force-pushed the 14.0-mail-tracking-email-deletion-job branch 3 times, most recently from 722eeeb to 6ed27f3 Compare August 14, 2024 11:04

henrybackman requested a review from sebalix August 14, 2024 12:03

sebalix reviewed Aug 19, 2024

View reviewed changes

mail_tracking/models/mail_tracking_email.py Outdated Show resolved Hide resolved

mail_tracking: add garbage collection

9fd6204

Add autovacuum to mail_tracking_email that removes old records based on new configuration variable mail_tracking_email_max_age_days. Due to possibly a large number of records to be deleted on first run, set a default limit of 5000 per run.

henrybackman force-pushed the 14.0-mail-tracking-email-deletion-job branch from 6ed27f3 to 9fd6204 Compare August 20, 2024 08:35

henrybackman requested a review from sebalix August 20, 2024 14:22

sebalix approved these changes Aug 26, 2024

View reviewed changes

simahawk approved these changes Aug 26, 2024

View reviewed changes

OCA-git-bot added the approved label Aug 26, 2024

pedrobaeza added this to the 14.0 milestone Aug 26, 2024

pedrobaeza approved these changes Aug 26, 2024

View reviewed changes

OCA-git-bot added the ready to merge label Aug 26, 2024

OCA-git-bot added the bot is merging ⏳ label Aug 26, 2024

OCA-git-bot merged commit 6bfbb1c into OCA:14.0 Aug 26, 2024
7 checks passed

OCA-git-bot added merged 🎉 and removed bot is merging ⏳ labels Aug 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[14.0][IMP] mail_tracking: add job to delete old tracking records #1430

[14.0][IMP] mail_tracking: add job to delete old tracking records #1430

henrybackman commented Aug 8, 2024 •

edited

Loading

pedrobaeza commented Aug 8, 2024

sebalix commented Aug 8, 2024

pedrobaeza commented Aug 8, 2024

henrybackman commented Aug 8, 2024

henrybackman commented Aug 9, 2024

simahawk commented Aug 9, 2024

sebalix Aug 13, 2024

simahawk commented Aug 26, 2024

pedrobaeza left a comment

OCA-git-bot commented Aug 26, 2024

OCA-git-bot commented Aug 26, 2024

OCA-git-bot commented Aug 26, 2024

-        records_to_delete = self.search(domain, limit=limit)
-        _logger.debug("Deleting %s mail.tracking.email records", len(records_to_delete))
-        return records_to_delete.unlink()
+        records_to_delete = self.search(domain, limit=limit).exists()
+        if records_to_delete:
+            _logger.info("Deleting %s mail.tracking.email records", len(records_to_delete))
+            self.flush()
+            query = "DELETE FROM mail_tracking_email WHERE id IN %s"
+            args = (tuple(records_to_delete.ids),)
+            self.env.cr.execute(query, args)
+            self.invalidate_cache()

[14.0][IMP] mail_tracking: add job to delete old tracking records #1430

[14.0][IMP] mail_tracking: add job to delete old tracking records #1430

Conversation

henrybackman commented Aug 8, 2024 • edited Loading

pedrobaeza commented Aug 8, 2024

sebalix commented Aug 8, 2024

pedrobaeza commented Aug 8, 2024

henrybackman commented Aug 8, 2024

henrybackman commented Aug 9, 2024

simahawk commented Aug 9, 2024

sebalix Aug 13, 2024

Choose a reason for hiding this comment

simahawk commented Aug 26, 2024

pedrobaeza left a comment

Choose a reason for hiding this comment

OCA-git-bot commented Aug 26, 2024

OCA-git-bot commented Aug 26, 2024

OCA-git-bot commented Aug 26, 2024

henrybackman commented Aug 8, 2024 •

edited

Loading